[AURON #2392] Support Spark 4.2 by XorSum · Pull Request #2404 · apache/auron

XorSum · 2026-07-20T06:09:49Z

Which issue does this PR close?

Closes #2392

Rationale for this change

https://spark.apache.org/releases/spark-release-4-2-0.html

What changes are included in this PR?

Are there any user-facing changes?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Yes
No

If yes, include: Generated-by: Kimi K3

ASF guidance: https://www.apache.org/legal/generative-tooling.html

XorSum · 2026-07-20T06:13:08Z

 /**
- * Base test class under org.apache.spark.sql to use package-private [[SQLTestUtils]]; extends
- * [[QueryTest]] for comparisons and checks.
+ * Base test class under org.apache.spark.sql to extends [[QueryTest]] for comparisons and checks.
+ * Before spark-4.1 also extends package-private [[org.apache.spark.sql.test.SQLTestUtils]].
 */
+@sparkverExcludeParents("4.2", "org.apache.spark.sql.test.SQLTestUtils")
 abstract class AuronQueryTest
    extends QueryTest
-    with SQLTestUtils
    with BeforeAndAfterEach
+    with org.apache.spark.sql.test.SQLTestUtils
    with AdaptiveSparkPlanHelper {


SQLTestUtils is deprecated since 4.2.0, and we should use QueryTest directly instead.

XorSum · 2026-07-20T06:21:58Z

+// org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
+private class ConfigEntryWithDefaultFunction[T](


In the Java file SparkAuronConfiguration.java, a Scala case class ConfigEntryWithDefaultFunction is invoked. However, SPARK-55928 added a new field to ConfigEntryWithDefaultFunction, which caused a compilation failure. Therefore, I copied the code of ConfigEntryWithDefaultFunction to make Auron compile successfully.

XorSum · 2026-07-20T06:30:15Z

      }
    }
+
+    def verExcludeParents(c: whitebox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = {


This macro verExcludeParents was AI-generated by Kimi K3. All other code was written by human.

slfan1989

LGTM overall, with just a couple of minor issues to address.

slfan1989 · 2026-08-01T23:03:12Z

 # -----------------------------------------------------------------------------
 SUPPORTED_OS_IMAGES=("centos7" "ubuntu24" "rockylinux8" "debian11" "azurelinux3")
-SUPPORTED_SPARK_VERSIONS=("3.0" "3.1" "3.2" "3.3" "3.4" "3.5" "4.0" "4.1")
+SUPPORTED_SPARK_VERSIONS=("3.0" "3.1" "3.2" "3.3" "3.4" "3.5" "4.0" "4.1" "4.2")


Thanks for adding Spark 4.2 support! Could you also update CONTRIBUTING.md? Its supported-version list and compatibility matrix currently stop at Spark 4.1, and the document requires them to be updated when support changes. Please add Spark 4.2 with Scala 2.13/JDK 17+ and note that Iceberg and Hudi are unsupported.

@slfan1989 Thanks for the review! I've updated both files

slfan1989 · 2026-08-01T23:04:06Z

+                </goals>
+                <configuration>
+                  <rules>
+                    <!-- Spark 4.1 requires JDK 17+ and Scala 2.13.x -->


Should this be Spark 4.2?

Co-authored-by: cxzl25 <3898450+cxzl25@users.noreply.github.com>

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds Spark 4.2 support across Auron’s Spark shims, build profiles, and CI pipelines.

Changes:

Introduces a new Maven build profile for spark-4.2 and updates build/CI matrices to include Spark 4.2.
Extends many @sparkver guards to include 4.2, plus adds new Spark 4.2-specific shim implementations where APIs diverged.
Adds a new macro annotation to conditionally exclude parent traits/classes and a local ConfigEntryWithDefaultFunction shim to replace a Spark-internal type.

Reviewed changes

Copilot reviewed 49 out of 49 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
spark-version-annotation-macros/src/main/scala/org/apache/auron/sparkver.scala	Adds `@sparkverExcludeParents` macro annotation to remove specific parents for selected Spark versions.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronShuffleDependency.scala	Extends Spark 4.x compatibility guard to include 4.2.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeBroadcastExchangeBase.scala	Extends Spark 4.x compatibility guard to include 4.2.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/columnar/AuronColumnarStruct.scala	Adds Spark 4.2 support and implements new `getBinaryView` override.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/columnar/AuronColumnarBatchRow.scala	Adds Spark 4.2 support and implements new `getBinaryView` override.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/columnar/AuronColumnarArray.scala	Adds Spark 4.2 support and implements new `getBinaryView` override.
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala	Updates pattern matches for aggregate expressions to accommodate Spark API changes.
spark-extension/src/main/scala/org/apache/spark/auron/spark/configurations/ConfigEntryHelper.scala	Adds a local replacement for Spark’s internal `ConfigEntryWithDefaultFunction`.
spark-extension/src/main/java/org/apache/auron/spark/configuration/SparkAuronConfiguration.java	Switches to the new local `ConfigEntryWithDefaultFunction` import.
spark-extension-shims-spark/src/test/scala/org/apache/spark/sql/AuronQueryTest.scala	Uses new macro annotation to exclude `SQLTestUtils` parent on Spark 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/joins/auron/plan/NativeSortMergeJoinExecProvider.scala	Extends provider support to include Spark 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/joins/auron/plan/NativeShuffledHashJoinExecProvider.scala	Extends provider support to include Spark 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/joins/auron/plan/NativeBroadcastJoinExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronShuffleWriter.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronShuffleManager.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronBlockStoreShuffleReader.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeWindowExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeUnionExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeTakeOrderedExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeSortExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeShuffleExchangeExec.scala	Extends Spark 4.x guards and Spark-wide guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeRenameColumnsExecProvider.scala	Extends provider support to include Spark 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeProjectExecProvider.scala	Extends provider support to include Spark 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativePartialTakeOrderedExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeParquetSinkExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeParquetInsertIntoHiveTableExec.scala	Extends Spark 4.x guards and Spark-wide guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeOrcSinkExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeOrcInsertIntoHiveTableExec.scala	Extends Spark 4.x guards and Spark-wide guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeLocalLimitExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeGlobalLimitExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeGenerateExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeFilterExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeExpandExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeCollectLimitExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeBroadcastExchangeExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeAggExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/execution/auron/plan/ConvertToNativeExec.scala	Extends Spark version guards to include 4.2.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/auron/ShimsImpl.scala	Adds `spark-4.2` shimVersion and a Spark 4.2-specific `copyBatchScanExecWithRuntimeFilters` implementation.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/auron/InterceptedValidateSparkPlan.scala	Extends Spark version guards to include 4.2.
pom.xml	Adds `spark-4.2` Maven profile with version properties and build enforcer rules.
auron-spark-ui/src/main/scala/org/apache/spark/sql/execution/ui/AuronAllExecutionsPage.scala	Extends Spark 4.x UI render guard to include 4.2.
auron-spark-tests/common/src/test/scala/org/apache/spark/sql/SparkExpressionTestsBase.scala	Extends Spark 4.x guards to include 4.2.
auron-build.sh	Adds Spark 4.2 to the supported build script versions list.
CONTRIBUTING.md	Documents Spark 4.2 as an available `--sparkver` and adds a row to the version matrix.
.github/workflows/tpcds.yml	Adds a Spark 4.2 TPC-DS CI job.
.github/workflows/tpcds-reusable.yml	Updates Spark distribution artifact selection logic to handle 4.2.
.github/workflows/build-macos-releases.yml	Adds Spark 4.2 to macOS release build matrix.
.github/workflows/build-arm-releases.yml	Adds Spark 4.2 to ARM release build matrix.
.github/workflows/build-amd64-releases.yml	Adds Spark 4.2 to amd64 release build matrix and includes Spark 4.2 JDK21 Scala 2.13 build.

Suppressed comments (3)

spark-version-annotation-macros/src/main/scala/org/apache/auron/sparkver.scala:1

This match on c.macroApplication is not exhaustive. If the macro application tree shape differs (e.g., due to compiler changes or different call sites), this will throw a MatchError at compile time. Add a default case that aborts with a clear message (similar to the class-only check later) so failures are deterministic and easier to diagnose.
spark-extension/src/main/scala/org/apache/spark/auron/spark/configurations/ConfigEntryHelper.scala:1
ConfigEntryWithDefaultFunction is declared private, but it is imported and used from Java (SparkAuronConfiguration.java). A top-level Scala private class is not accessible from other compilation units, which will break compilation. Make this class accessible (e.g., remove private or scope it with an appropriate qualifier like private[configurations]), and ensure the Java import matches the chosen visibility.
spark-extension-shims-spark/src/test/scala/org/apache/spark/sql/AuronQueryTest.scala:1
The Scaladoc has a grammatical error ('to extends' → 'to extend') and the version statement ('Before spark-4.1 also extends...') doesn’t align with the annotation, which excludes the parent only for Spark 4.2. Either adjust the comment to match the actual behavior, or widen/narrow the @sparkverExcludeParents version selector so it matches the documented Spark version boundary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

 | 3.5 | 2.12 | 8+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | 0.15 |
 | 4.0 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | — |
 | 4.1 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | — | — |
+| 4.2 | 2.13 | 17+ | `rockylinux8` | - | - | - | - | — | — |


github-actions Bot added infra spark build spark-ui spark-tests labels Jul 20, 2026

XorSum commented Jul 20, 2026

View reviewed changes

cxzl25 reviewed Jul 20, 2026

View reviewed changes

Comment thread .github/workflows/build-amd64-releases.yml Outdated

slfan1989 requested changes Aug 1, 2026

View reviewed changes

github-actions Bot added the documentation Improvements or additions to documentation label Aug 2, 2026

XorSum and others added 3 commits August 3, 2026 13:43

Support Spark 4.2.0

5a9c3d3

Update .github/workflows/build-amd64-releases.yml

8d072a7

Co-authored-by: cxzl25 <3898450+cxzl25@users.noreply.github.com>

update CONTRIBUTING.md and pom.xml

fc6eb77

XorSum force-pushed the support-spark-4.2.0 branch from 8106192 to fc6eb77 Compare August 3, 2026 06:21

slfan1989 approved these changes Aug 4, 2026

View reviewed changes

SteNicholas requested a lite review from Copilot August 5, 2026 09:08

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Copilot started reviewing on behalf of SteNicholas August 5, 2026 09:49 View session

Navigation Menu

Uh oh!

[AURON #2392] Support Spark 4.2 - #2404

[AURON #2392] Support Spark 4.2#2404
XorSum wants to merge 3 commits into
apache:masterfrom
XorSum:support-spark-4.2.0

XorSum commented Jul 20, 2026

Uh oh!

XorSum Jul 20, 2026

Uh oh!

XorSum Jul 20, 2026

Uh oh!

XorSum Jul 20, 2026

Uh oh!

Uh oh!

slfan1989 left a comment

Uh oh!

slfan1989 Aug 1, 2026

Uh oh!

XorSum Aug 2, 2026

Uh oh!

slfan1989 Aug 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
		private class ConfigEntryWithDefaultFunction[T](

Uh oh!

Conversation

XorSum commented Jul 20, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

XorSum Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

XorSum Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

XorSum Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

slfan1989 left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

XorSum Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

slfan1989 Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants